Relaxations for inference in restricted Boltzmann machines

We propose a randomized relax-and-round inference algorithm that samples near-MAP configurations of a binary pairwise Markov random field. We experiment on MAP inference tasks in several restricted Boltzmann machines. We also use our underlying sampler to estimate the log-partition function of restricted Boltzmann machines and compare against other sampling-based methods. 1. Background and setup A binary pairwise Markov random field (MRF) over n variables x ∈ {0, 1} models a probability distribution pÃ(x) ∝ exp(xÃx). The non-diagonal entries of the matrix à ∈ Rn×n encode pairwise potentials between variables while its diagonal entries encode unary potentials. The exponentiated linear term xÃx is the negative energy or simply the score of the MRF. A restricted Boltzmann machine (RBM) is a particular MRF whose variables are split into two classes, visible and hidden, and in which intra-class pairwise potentials are disallowed. Notation We write Symn for the set of symmetric n× n real matrices, and S to denote the unit sphere {x ∈ R : ‖x‖2 = 1}. All vectors are columns unless stated otherwise. 1.1. Integer quadratic programming Finding the maximum a posteriori (MAP) value of a discrete pairwise MRF can be cast as an integer quadratic program (IQP) given by max x∈{−1,1}n xAx (1) International Conference on Learning Representations, Banff, Canada, 2014. ∗Authors contributed equally. Note that we have the domain constraint x ∈ {−1, 1} rather than {0, 1}. We relate the two in Section 2.3.